Adherence of SARS‐CoV‐2 Seroepidemiologic Studies to the ROSES‐S Reporting Guideline During the COVID‐19 Pandemic

ABSTRACT Background Complete reporting of seroepidemiologic studies is critical to their utility in evidence synthesis and public health decision making. The Reporting of Seroepidemiologic studies—SARS‐CoV‐2 (ROSES‐S) guideline is a checklist that aims to improve reporting in SARS‐CoV‐2 seroepidemiologic studies. Adherence to the ROSES‐S guideline has not yet been evaluated. Objectives This study aims to evaluate the completeness of SARS‐CoV‐2 seroepidemiologic study reporting by the ROSES‐S guideline during the COVID‐19 pandemic, determine whether guideline publication was associated with reporting completeness, and identify study characteristics associated with reporting completeness. Methods A random sample from the SeroTracker living systematic review database was evaluated. For each reporting item in the guideline, the percentage of studies that were adherent was calculated, as well as median and interquartile range (IQR) adherence across all items and by item domain. Beta regression analyses were used to evaluate predictors of adherence to ROSES‐S. Results One hundred and ninety‐nine studies were analyzed. Median adherence was 48.1% (IQR 40.0%–55.2%) per study, with overall adherence ranging from 8.8% to 72.7%. The laboratory methods domain had the lowest median adherence (33.3% [IQR 25.0%–41.7%]). The discussion domain had the highest median adherence (75.0% [IQR 50.0%–100.0%]). Reporting adherence to ROSES‐S before and after guideline publication did not significantly change. Publication source (p < 0.001), study risk of bias (p = 0.001), and sampling method (p = 0.004) were significantly associated with adherence. Conclusions Completeness of reporting in SARS‐CoV‐2 seroepidemiologic studies was suboptimal. Publication of the ROSES‐S guideline was not associated with changes in reporting practices. Authors should improve adherence to the ROSES‐S guideline with support from stakeholders.

Conclusions: Completeness of reporting in SARS-CoV-2 seroepidemiologic studies was suboptimal.Publication of the ROSES-S guideline was not associated with changes in reporting practices.Authors should improve adherence to the ROSES-S guideline with support from stakeholders.

| Introduction
Measuring the prevalence of antibodies against SARS-CoV-2 in population blood and serum samples (seroprevalence) provides valuable estimates of true population infection, including infections in asymptomatic individuals or those who were unable to acquire diagnostic services [1].This makes serosurveillance a valuable tool for pandemic decision-making.Seroepidemiologic studies must be well reported to be useful, however, as information about how seroprevalence was measured (e.g., immunoassay characteristics and validation), sampling methods, and statistical techniques affects study interpretation, comparison, and application [2].
Inconsistent reporting of study methods and results was evident during the first year of the COVID-19 pandemic [3] despite the existence of guidelines for study designs typically used to estimate seroprevalence (e.g., the "Strengthening the Reporting of Observational Studies in Epidemiology" [STROBE] guidelines) [4].Reporting guidelines were also available for specific domains of epidemiological research, such as observational infectious disease studies (e.g., the "Strengthening the Reporting of Molecular Epidemiology for Infectious Diseases" [STROME-ID] statement) and influenza seroepidemiology (e.g., the CONSISE "Reporting Of Sero-Epidemiologic Studies for Influenza" [ROSES-I]) statement.Previous studies suggest that the use of these types of guidelines improves reporting [5,6], yet adherence to such guidelines still appears suboptimal.For instance, estimates of adherence to STROBE's 22 reporting criteria have ranged from 51.4% to 76.5% [7][8][9][10], with similar estimates of adherence (50%) to STROME-ID [11].
In June 2021, the World Health Organization (WHO) published the "Reporting of Sero-Epidemiologic Studies-SARS-CoV-2" (ROSES-S) guideline as a content-specific extension of the STROBE checklist.ROSES-S is a checklist of 22 criteria for the reporting of SARS-CoV-2 seroepidemiologic studies, regardless of study design [12].
To better understand reporting completeness among SARS-CoV-2 seroepidemiologic studies and determine whether ROSES-S has changed practice, we conducted a sub-study within a living systematic review with the following aims: (i) quantify the completeness of reporting among SARS-CoV-2 seroepidemiologic studies by assessing adherence to the ROSES-S reporting guideline; (ii) assess whether specific study characteristics were associated with adherence to the ROSES-S guideline; and (iii) determine whether implementation of the ROSES-S guideline was associated with changes in the reporting completeness of SARS-CoV-2 seroepidemiologic studies.

| Search Strategy and Eligibility Criteria
This systematic review sub-study was reported according to the Preferred Reporting Items for Systematic Review and Meta Analysis (PRISMA) Statement [29] (Data S1) and was embedded within SeroTracker's living global systematic review of SARS-CoV-2 seroepidemiologic studies (PROSPERO [CRD42020183634]) [13], which is available on a public dashboard and data platform [14].An in-depth description of the methods, including a full search strategy, inclusion and exclusion criteria, and screening process, has been previously published [14,15].Briefly, a search of electronic databases, gray literature, and news media captured published and pre-print studies, as well as institutional reports from January 1, 2020, to December 31, 2022.Direct submissions of data were also invited at SeroTracker.com.
A seroepidemiologic study was defined as estimating the prevalence of SARS-CoV-2 antibodies in humans through the collection and testing of serum (or proxy, such as oral fluid) specimens from a defined sample frame over a specified period of time.Eligible study designs included cross sectional studies, repeated cross-sectional studies, and cohort studies.Studies of all languages were included if they reported the number of participants, sampling end date and week, geographic location of sampling, and a prevalence estimate.Studies in languages other than English, French, Spanish, Romanian, Persian, and Chinese were translated via Google translate machine translation.
Studies were screened using Covidence systematic review manager [16] via a two-stage screening process by two independent reviewers.The first stage determined inclusion based on review of study title and/or abstract.The second stage determined inclusion by review of the full text.A third team member resolved discrepancies by arbitration.

| Study Selection
We included a random time-stratified sample (n = 199) of peerreviewed, pre-print studies, and gray literature (government and institutional reports) from March 2020 to December 2022 to capture studies before and after the publication of the ROSES-S guideline (June 2021).The sample size was determined based on a precision level of ±7% with 95% confidence and a 60% estimated adherence to checklist items [11,17].To ensure a comparable sample size before and after the ROSES-S guideline's publication date, we created strata of six month discretized time periods and randomly sampled 34 studies from each strata.If a given month had only one included study, then we conducted additional random sampling of studies from that month to select an additional study-bringing the total to two.

| Data Extraction
The following data was extracted by one independent reviewer using an Airtable spreadsheet tool (2022) and verified by a second reviewer from the SeroTracker team: article publication source, study design, publication month, sampling frame, sampling method, sample size, geographic scope (local, regional, national), WHO Humanitarian Response Plan (HRP) status [18], WHO Unity protocol alignment [19], journal impact factor (IF), SARS-CoV-2 antibody seroprevalence estimate(s), and whether the article cited the ROSES-S guideline or other reporting guidelines.The risk of bias (RoB) (low, moderate, high) was evaluated by two independent reviewers using a modified version of the Joanna Briggs Institute checklist [20,21].Further details on extraction methods can be found in a previous publication [22].

| Reporting Guideline Adherence Scoring System
The ROSES-S guideline scoring sheet split each of the 22 checklist items into their component paragraphs, with each paragraph containing specific and related reporting criteria (Data S2).As a result, a total of 68 items were scored independently.Only the second criterion of item 5 ("Describe the timing of the biological sampling in relation to the disease epidemiology in the study population [the beginning, peak, and end of virus transmission], Describe any vaccination efforts that have been undertaken") was further split into two items (items 5.2 and 5.3), as vaccination was not relevant in the beginning of the pandemic.
To ensure inter-rater agreement, raters (AS, BC, EL) piloted 10 eligible studies; discordances were discussed with arbitration by a co-creator of the ROSES-S guideline (NB).Scoring was completed by categorizing each statement as "reported" or "not reported."A "not applicable" (N/A) option was possible for 22/68 (32.4%) items related to specific study designs or that were not relevant at all stages of the pandemic.For example, items pertaining to case-control studies were automatically "not applicable" given that the case-control study design was not eligible for inclusion in our study.N/A scores were not assigned a numeric value and, therefore, were not included in the denominator when calculating adherence proportions.Where there were multiple criterion embedded in a single reporting item that was not further broken down in the scoring sheet, the item was scored as "reported" if any of the criteria were satisfied.For each study, the adherence scores for each of the six domains were calculated, as well as an overall adherence score across all items.

| Statistical Analysis
The study characteristics were summarized using descriptive statistics.We calculated the percentage of studies adhering to each item of the ROSES-S scoring sheet, as well as the median percentage and interquartile range (IQR) of adherence across all items and for each domain (i.e., title, abstract, and introduction; epidemiological methods; laboratory methods; results; discussion; other information) and for each study characteristic (article publication source, study design, publication month, sampling frame, sampling method, sample size, overall RoB, geographic scope, WHO HRP status, WHO Unity protocol alignment, IF, and whether the study cited ROSES-S or other reporting guidelines).Overall percentage of adherence and percentage adherence by domain were visualized with violin plots, and the relationship between IF and adherence was visually inspected using a scatter plot.A publication time lag of 154 days was used; this was the estimated median time between the last date of participant sampling and the publication date of SARS-CoV-2 seroepidemiologic studies during that time period, irrespective of publication type [23].Three time periods were used: pre-ROSES-S publication (March 1, 2020-June 26, 2021), 0-154 days post-ROSES-S publication (June 27, 2021-November 27, 2021), and 155-553 days post-ROSES-S publication (November 28, 2021-December 31, 2022).The RoB evaluations were summated using percentages.
Beta regression was used to investigate the association of study characteristics and publication of the ROSES-S guideline with total adherence scores.A univariable continuous piecewise model with two knots-one at the date of ROSES-S publication and the other 154 days later-was constructed to assess the total adherence scores over time and was plotted to conduct a visual inspection of trends.A multivariable model was also built to explore all candidate study characteristic predictors.Marginal effects were calculated for the final model to facilitate the interpretation of model coefficients.All adherence scores and predicted score changes from the beta regression models are reported as percentages.See Data S2 for more details.Analysis was conducted using the betareg (version [v] 3.1.4)[24], lmtest (v 0.9.40) [25], MuMIn (v 1.47.5)[26], and mfx (v 1.2.2) [27] packages in R (v 4.2.2).

| Results
One hundred and ninety-nine articles were included and analyzed (Figure A1; Data S2).Study characteristics are reported in Table 1.Included studies were mostly published in peerreviewed journals (68.3%), used one-time cross-sectional survey design (74.9%), and were sampled from the general population (52.8%).The studies mostly used a non-probability sampling method (71.3%) and incorporated a sample size of 1000+ participants (46.2%).The majority of studies were local in geographic scope (61.3%) and originated from geographic locations without HRP status (81.4%).Most studies were not aligned with the WHO Unity protocol (77.9%) and did not cite any reporting guidelines (97%), with none citing ROSES-S.Lastly, articles were assessed to have a high RoB (61.8%), with RoB item 8b ("Was there appropriate adjustment for population characteristics?") the least fulfilled (12.6%) (Table A1).
The adherence of study reporting to ROSES-S items is reported in Table 2.The median adherence to ROSES-S reporting items was 48.1% (IQR 40.0%-55.2%)per study.Adherence ranged from 8.8% to 72.7% per study.Overall, the most frequently reported ROSES-S items were "State the interval between sequential biological samples (serial cross-sectional or longitudinal studies), or specify whether only a single sample was collected (cross-sectional study)" (item 5.5: 197/199 [99.0%]), and "For a cross-sectional study, report the numbers of outcome events or summary measures" (item 15.3: 160/163 [98.2%]).The least frequently reported ROSES-S items were "For a cohort study, explain how loss to follow-up was addressed, if applicable" (item 12.5: 0/14 [0%]) and "Specify laboratory biosafety conditions" (item 13.10): 2/198 [1.0%]).
Median adherence to the ROSES-S items was similar before and after the publication of the ROSES-S guideline (Table 2).Median adherence was 47.3% (IQR 38.5%-55.6%)prior to ROSES-S publication, 50.9% (IQR 45.5%-55.2%)0-154 days after publication, and 46.6% (IQR 39.6%-54.6%)155-553 days after publication.Adherence to items fluctuated over time, increasing for some items after the publication of the ROSES-S guideline and decreasing for others.For example, reporting of "For a cross-sectional study, describe analytical methods taking account of sampling strategy, if applicable" (item 12.7) increased from 17.3% (13/75 studies) pre-ROSES-S to 60.8% (31/51 studies) 155-553 days post-ROSES-S.In contrast, reporting of "Describe illness definitions and methods for ascertaining the presence or absence of clinical illness in subjects" (item 7.5) decreased from 65.1% (56/86 studies) pre-ROSES-S to 40% (24/60 studies) 155-553 days post-ROSES-S.Piecewise time series analysis showed no significant changes in reporting after the publication of the ROSES-S guideline (Figure 1).Multivariable analysis for beta regression is presented in Table 3. Overall reporting adherence was significantly associated with article publication source, study RoB, and sampling method.Institutional reports and conference abstracts or news articles had considerably lower mean total adherence to ROSES-S compared to published peer-reviewed journal articles, respectively, at −14.0% (−19.4% to −8.6%) and −29% (−35.1% to −22.9%).Mean total adherence to ROSES-S was 5.8% (2.3%-9.3%)and 7.8% (0.3%-15.2%) higher for studies with moderate and low RoB, respectively, compared to those with a high RoB.Furthermore, studies that used an unclear sampling method had a −9.4% (15.8% to −2.9%) lower mean total adherence to ROSES-S compared to those that used a non-probability sampling method.Univariable beta regression also showed that special population studies had higher adherence scores compared to general population studies with an increase of 4.2% (0.5%-7.8%) (Table A3).
Lastly, IF and adherence score were plotted for included peerreviewed studies with available IFs (N = 133) to assess whether   there was an association between quality of manuscript and adherence score, using IF as a proxy for quality (Figure A3).The mean IF across studies was 7.18.Upon visual inspection, there did not appear to be a relationship between IF and adherence score.The fitted univariable beta regression did not show a significant change in average adherence score with one unit increase in IF [0.01%(−0.21%-0.23%)](Table A3).

| Discussion
The ROSES-S guideline was published to facilitate the reporting of a minimal collection of information in SARS-CoV-2 seroepidemiologic studies.This systematic review sub-study examined the completeness and associated factors of reporting in SARS-CoV-2 seroepidemiologic studies published in the first two years of the COVID-19 pandemic as determined by adherence to the ROSES-S guideline.Reporting completeness was found to be suboptimal, which may have implications for consumers of said scientific literature in assessing study quality.The median adherence to ROSES-S items was 48.1% per study, which is comparable to estimates from studies evaluating adherence to STROBE and STROME-ID [7][8][9][10][11].
There was variation in the completeness of reporting for different reporting items.The domains of the ROSES-S guideline with the least frequently reported items were laboratory methods and results, and the least reported individual items were related to serological assays and statistical methods.This aligns with evaluations of STROBE, where key methodological items seem to be consistently underreported (e.g., sample size estimation, addressing missing data, addressing loss to follow-up) [28].This trend persists in evidence synthesis citing more heavily journal endorsed and established guidelines, such as the PRISMA
The publication of the ROSES-S guideline was not associated with a change in reporting completeness, suggesting that the guideline has not had an impact on reporting practices to date.Other studies using segmented regression to evaluate the impact of the STROBE guidelines publication on reporting have similarly found no significant association [8].This contrasts with evaluations of the "Consolidated Standards of Reporting Trials" (CONSORT) guideline [31], which was originally published in 1996 and last updated in 2001 [32], as well as the PRISMA statement.According  to PRISMA, from publication in 2010 to 2016, item reporting adherence increased in systematic reviews, compared to systematic reviews published from 1989 to 2016 [30].This suggests that a longer follow-up time may be needed to increase awareness and uptake of the ROSES-S guideline, which was published in 2021.This is reinforced by our finding that only 3.7% of peer-reviewed studies cited a reporting guideline, none of which were ROSES-S, suggesting a lack of awareness of the ROSES-S guideline.
Complete reporting of methodological items is necessary to inform appropriate adjustments of estimates based on differences in study methodology.Methodological heterogeneity has been observed in previous syntheses of SARS-CoV-2 seroepidemiologic studies and has been associated with notable differences in estimates [30,31].For example, in one study, failing to account for methodological differences between serosurveys led to variation in estimates as high as 16% [31].In instances where methods are not clearly reported, adjustment may not be possible, and this may affect the accuracy, comparability, and generalizability of synthesized seroprevalence estimates.
Complete reporting of items related to sampling and measurement methods is particularly important given their impact on study RoB [21].ROSES-S items, such as those related to sample recruitment, size, and retention, as well as immunoassay validation and testing algorithm, inform judgments of selection bias and measurement bias, respectively [21].For instance, incomplete reporting of immunoassay validation details may contribute to misleading results, as it limits the opportunity for adjustment of seroprevalence estimates for poor test performance, which has been shown to heavily influence seroprevalence estimates [31].Incomplete reporting thus directly affects RoB assessment and creates the potential for misleading meta-analyzed estimates, hindering the utility of individual studies and evidence synthesis.
Journal IF is an important metric to assess the visibility and reputation of a journal.It was used as a proxy for quality to investigate the association between quality of manuscript and adherence score.Because the results of this study did not demonstrate that IF had any relationship with or influence on adherence score, it can be deduced that IF does not reflect the reporting completeness of individual studies.
Interventions to improve adherence to reporting guidelines have been proposed and tested.For instance, there could be greater encouragement of use of the guidelines, such as integrating checklist writing tools in the submission process (e.g., COBWEB), which has improved overall reporting completeness for randomized controlled trials [33].Although journal endorsement of reporting guidelines seems to impact adherence [6,32,34], it can be ineffective without enforcement.Verifying adherence to a guideline throughout the peer-review process could also be considered.Editors may ask authors to flag nonadherence to a given guideline.When such an intervention was evaluated using the CONSORT guideline, improvements in reporting were demonstrated [35].Peer-review tools, such as StatReviewer, that automatically check guideline adherence against manuscripts may also improve adherence [36].
There are other potential solutions specifically tailored to the reporting of serosurveillance data.For instance, reporting guidelines could be included in epidemiological study protocols, like the Unity protocol for COVID-19 [19], to increase awareness, and to place emphasis on methodological quality and reporting completeness at the outset of conducting a study.Reporting completeness and guidelines should also be considered by stakeholders establishing and maintaining serological surveillance networks in preparation for the next pandemic.
We found that institutional reports, conference abstracts, and news and media reports had lower ROSES-S adherence compared to peer-reviewed journal articles.This may in part be attributable to limited reporting space in shorter form publication outlets like media reports and conference abstracts.Although the ROSES-S guideline was designed for full length seroepidemiologic study articles, gray literature including news media or public health reports can be used in evidence synthesis and, therefore, may warrant development of an adapted guideline for short form articles.Meanwhile, the lack of publication standards and peer-review for institutional reports may be contributing factors for poor reporting.The existing ROSES-S guideline is a suitable resource for those types of articles and could be better utilized.
Our results showed that RoB was associated with adherence to the ROSES-S guideline, similar to other studies that have found correlations between lower RoB and reporting completeness [37,38].This association may arise from differences in the behaviors, traits, or circumstances of investigators that produce studies of higher versus lower methodological quality.For example, investigators conducting studies of higher methodological quality may have more resources and time to produce comprehensive studies, or more training and knowledge about study methods, including reporting.If so, enhanced training or additional resources for investigators conducting seroepidemiologic studies may be needed to enable comprehensive reporting (e.g., user-friendly templates).
This study had two key strengths.Firstly, to the best of our knowledge, this was the first analysis to evaluate the reporting completeness of SARS-CoV-2 seroepidemiologic studies according to the ROSES-S guideline.Secondly, we sampled from the largest global repository of SARS-CoV-2 seroepidemiologic studies that was developed using a robust systematic review; thus, the results likely have good generalizability.
This study had several limitations.Firstly, the 154-day median publication delay time utilized in the analysis may not have been sufficiently long.Articles published after December 2021 may have been submitted before publication of the ROSES-S guideline and, therefore, subject to misclassification bias as the guideline was not available when the authors were reporting their data.However, our sensitivity analysis using a longer time delay (255 days, the third quartile of previously cited median publication lag time) [23] did not differ from the main analysis (Table A3).Longer delays to publication are possible, but likely only affected a small number of studies [23].Secondly, as some reporting criterion needed to be reported together to be interpretable, we may have overestimated adherence to multiconditional items given that we scored the item as "reported" if any criterion were satisfied.Thirdly, decisions on whether a reporting item was achieved were subjective and other investigators may evaluate adherence differently.However, we mitigated this by developing our scoring sheet items in consultation with developers of the ROSES-S guideline, piloted our scoring sheet, and discussed discrepancies to ensure inter-rater agreement.
Examining study quality is an aim for our future work, building upon the knowledge gained through this review and others, and would consider the influence of different variables before classifying a manuscript as either good or bad quality.
In summary, this study found that the completeness of reporting in SARS-CoV-2 seroepidemiologic studies was suboptimal and associated with key study characteristics including article source of publication, RoB, and sampling method.The publication of the ROSES-S guideline was not associated with changes in reporting practices.Improvements in the completeness of reporting may serve to increase the transparency, comparability, and reproducibility of seroepidemiologic research.Authors, journal editors, funders, decision-makers, and other stakeholders in infectious disease research and public health should consider approaches to facilitate guideline adherence such as ROSES-S guideline endorsement and promotion, journal policies on reporting, and other potential funding incentives.

FIGURE 1 |
FIGURE 1 | Adherence to ROSES-S over time.Each dot represents a study.Dashed lines are the 2.5% and 97.5% quantiles of the predicted beta distribution.This piecewise model was not statistically significant compared to non-segmented [X 2 (2) = 2.68, p = 0.262] and null [X 2 (3) = 2.85, p = 0.415] models in likelihood ratio tests.

FIGURE A2 |
FIGURE A2 | Adherence to ROSES-S (%) overall and per domain.For all violin plots, n = 199, except for Domain 6, for which n = 197.
Abbreviation: HRP, Humanitarian Response Plan. a Publication date is a continuous variable with monthly increments.Each month is considered equal to 30.4 days.b General population includes the following sampling frames: household and community samples, blood donors, residual sera, persons living in slums, pregnant or parturient women, and representative patient populations.c None cited ROSES-S.The only cited guidelines were STROBE (n = 5) and TREND (n = 1).d Only studies published in a peer-reviewed journal with a journal impact factor (n = 133) were included in this model.*p < 0.05, **p < 0.01, and ***p < 0.001.
a General population includes the following sampling frames: household and community samples, blood donors, residual sera, persons living in slums, pregnant or parturient women, and representative patient populations.b WHO Unity-aligned: studies aligned with the WHO Unity protocol.

TABLE 2 |
Adherence to ROSES-S overall and pre/post ROSES-S publication.

TABLE 3 |
Multivariable beta regression identifying factors associated with reporting completeness.

TABLE A2 |
Median and IQR adherence scores of study characteristics stratified by ROSES-S publication time period.

TABLE A3 |
Univariable beta regression separately evaluating the association of each candidate predictor with total adherence score.